The invention relates to a method and a corresponding apparatus for segmenting images of a movable camera and for determining the pose of a movable camera.
Wearable apparatuses for augmenting the perception of reality of a user, such as, e.g. glasses with overlays with respect to the surroundings observed through the glasses, may also be used in a vehicle, for example to provide an occupant of the vehicle with more in-depth information with respect to traffic, more in-depth information with respect to the interior of the vehicle and/or warning notices. Here, where possible, information and/or notices which are connected to one or more objects in fact observed by the user of the apparatus should be output.
An apparatus for augmenting the perception of reality, e.g. smartglasses, typically includes a camera which captures surroundings observed by the user. The camera usually has a movable arrangement such that different surroundings observed by the user may be captured. By way of example, smartglasses typically have a camera which is able to capture surroundings of the smartglasses. This camera moves with the head of the user of the smartglasses.
In order to be able to uniquely position and identify an object in the observed surroundings of the user on the basis of image data captured by the camera, the pose of the camera, i.e. the position of the camera in 3-dimensional (3D) space and the orientation of the camera in 3D space, typically needs to be determined first. The pose of the camera may be determined on the basis of the detection of one or more known objects within the image data. More particularly, known features (e.g. feature points) or objects in the interior of a vehicle (e.g. a known CAD (computer-aided design) model of the interior of the vehicle) may be used to determine the pose of the camera.
The present document considers the technical problem of determining the pose of a movable camera in a vehicle with high accuracy. In this context, the present document considers, more particularly, the technical problem of precisely segmenting images captured by the movable camera into a foreground region and into a background region. The foreground region (which primarily comprises the interior of the vehicle) may then be used for determining the pose of the camera.
In accordance with one aspect of the invention, a method is described for segmenting a first scene (or a first image) from a sequence of scenes (or from a sequence of images). More particularly, the method is directed to subdividing the first scene into a foreground region and into a background region. In other words, pixels or image points of the first scene should be assigned either to a foreground region or to a background region. Here, the foreground region may primarily comprise the interior of a vehicle and the background region may primarily comprise exterior surroundings of the vehicle. In other words, the foreground region may comprise elements/objects (e.g. parts of the vehicle) which move with the vehicle. On the other hand, the background region may comprise elements/objects which do not move with the vehicle (e.g. exterior surroundings of the vehicle).
The sequence of scenes was captured by a camera (e.g. a mono camera or a stereo camera) of a vehicle, for example by a camera in or on a vehicle. The vehicle comprises e.g. a road vehicle, for example an automobile or a truck. The camera may be movably arranged in the interior of the vehicle and/or behind the windshield of a vehicle (e.g. of a motor bike). By way of example, the camera may be arranged on the head of a driver of the vehicle (e.g. on smartglasses worn by the driver of the vehicle). The camera may be configured to capture scenes or images which include both part of the interior of the vehicle (or part of the vehicle) and part of exterior surroundings of the vehicle. By way of example, the exterior surroundings of the vehicle may be captured through a window (e.g. through a windshield) of the vehicle. Therefore, the first scene may comprise an interior portion, in which an interior of the vehicle and/or parts of the vehicle are displayed and an exterior portion, in which external surroundings of the vehicle are displayed.
The method includes determining an optical flow, more particularly optical flow vectors, of the first scene on the basis of the sequence of scenes. Here the optical flow, more particularly the optical flow vectors, of the first scene can be determined for a multiplicity of features (or for a multiplicity of prominent image points) of the first scene. More particularly, the optical flow, more particularly the flow vectors, may be determined for so-called ORB (Oriented FAST and Rotated BRIEF) features. Here, the optical flow, more particularly the optical flow vectors, of the first scene may be determined in an efficient and robust manner by use of a hierarchic procedure, for example by use of the hierarchic Lucas-Kanade method. By way of example, the optical flow, more particularly the optical flow vectors, of the first scene may be determined in a hierarchic fashion for different resolutions of the multiplicity of features. Here, the resolution of the multiplicity of features may be successively increased.
The method further comprises determining statistical variables of the optical flow, i.e., more particularly, of the optical flow vectors, in different portions of the first scene. The statistical variable of optical flow vectors in a portion may e.g. comprise a mean magnitude or a mean size/strength of the optical flow vectors in the portion. In an alternative or complementary manner, the statistical variable may comprise a measure of homogeneity (e.g. a variance or a standard deviation) of the optical flow vectors in the portion. By way of example, the portions may correspond to a preliminary or final background region and foreground region of the first scene.
Moreover, the method comprises determining a separating line between a foreground region and a background region of the first scene on the basis of the statistical variables and, optionally, on the basis of the different portions. In other words, the curve of a separating line which subdivides the first scene into a foreground region and into a background region may be determined on the basis of the statistical variables. Here, determining the separating line comprises determining, on the basis of the statistical variable, an indicator that, whether, to what extent and/or how far the optical flow, i.e., more particularly, the optical flow vectors, in the background region can be traced back to a movement of the vehicle.
Hence, on the basis of analyzing the optical flow (i.e., more particularly, the flow vectors) and on the basis of the fact that some of the flow vectors (i.e., more particularly, the flow vectors which may be traced back to features in the external portion of the first scene) are caused by the movement of the vehicle, the method facilitates an efficient and robust segmentation of the scene into a foreground region and into a background region. By way of example, this segmentation may be used to precisely determine the pose of the e.g. movable camera in the vehicle interior.
The indicator that, whether, to what extent and/or how far the optical flow, i.e., in particular, the flow vectors, in the background region can be traced back to a movement of the vehicle may comprise a measure of distance which indicates a distance of a statistical variable of the optical flow vectors in the foreground region from a corresponding statistical variable of the optical flow vectors in the background region. By way of example, the separating line may be determined in such a way that the measure of distance is increased (possibly maximized). The use of a measure of distance as an indicator for whether the flow vectors in the background region can be traced back to a movement of the vehicle is based on the observation that the statistical behavior of the flow vectors in the interior of the vehicle or at parts of the vehicle deviates significantly from the statistical behavior of the flow vectors in the exterior surroundings of the vehicle. By classifying the flow vectors and the corresponding image points depending on the measure of distance, it is possible to bring about a robust and precise division of the first scene into a foreground region and into a background region.
The measure of difference may comprise a difference between the corresponding statistical variables. By way of example, the measure of distance may comprise a difference between the mean magnitude of the optical flow vectors in the (possibly preliminary) foreground region and the mean magnitude of the optical flow vectors in the (possibly preliminary) background region. This measure of distance exploits the fact that, in the case of the vehicle driving, the flow vectors in the exterior portion of the first scene typically have a greater magnitude than in the interior portion. Hence, the measure of distance (in particular, a value of the measure of distance) is an indicator that or whether or to what extent the flow vectors of the (possibly preliminary) background region can be traced back to a movement of the vehicle.
Alternatively, or in a complementary manner, the measure of distance may comprise a difference between the measure of homogeneity of the optical flow vectors in the (possibly preliminary) foreground region and the measure of homogeneity of the optical flow vectors in the (possibly preliminary) background region. This measure of distance exploits the fact that, when the vehicle is driving, the flow vectors in the external portion of the first scene are typically substantially less homogeneous than in the interior portion. Hence, the measure of distance (in particular, a value of the measure of distance) is an indicator that or whether or to what extent the flow vectors of the (possibly preliminary) background region can be traced back to a movement of the vehicle.
The measure of distance may e.g. comprise a combination (e.g. a weighted sum) of the aforementioned differences. The accuracy and robustness of a segmentation may be further improved by such a combination.
The method may comprise determining a preliminary curve of the separating line. On the basis of the preliminary curve of the separating line, the first scene may be subdivided into a preliminary foreground region and into a preliminary background region. Then, it is possible to determine a statistical variable of the optical flow vectors in the preliminary foreground region and a corresponding statistical variable of the optical flow vectors in the preliminary background region. Moreover, it is possible to determine a value of the indicator on the basis of the determined statistical variables for the preliminary foreground region and for the preliminary background region. The curve of the separating line may then be fitted in such a way that the value of the indicator is improved (possibly optimized, i.e. maximized or minimized). The (final) separating line between the (final) foreground region and the (final) background region may, in particular, be determined in iterative fashion (by repeating the aforementioned steps). Here, the indicator may be used as an optimization criterion and a value of the indicator (i.e. a value of the optimization criterion) may be improved (possibly optimized) by the iterations. Hence, there may be a robust segmentation of the first scene.
One or more conditions with respect to the separating line may be taken into account when determining the separating line. The efficiency and the robustness of the method may be increased by conditions. The conditions allow the fact that a separation between windows (i.e. the external portion of the first scene) and interior (i.e. internal portion of the first scene) typically extends horizontally in vehicles to be taken into account. By way of example, the separating line may be determined in such a way that the separating line has exactly one point on a vertical axis of the first scene for each point on a horizontal axis of the first scene. In other words, the separating line may be determined in such a way that the separating line assumes exactly one value on the vertical axis for each value on the horizontal axis. Furthermore, the separating line may be determined in such a way that the value of a mean gradient of the separating line along the horizontal axis is less than or equal to a predefined gradient threshold. By way of example, the gradient threshold may lie at zero. Thus, it is possible to determine a separating line which, on average, extends horizontally through the first scene. Secondly, when determining the separating line, it is possible to stipulate that the separating line extends horizontally through the first scene in the entire region. Moreover, it is possible to stipulate that the curve of the separating line is continuous, at least in portions.
The method may further comprise determining a driving speed of the vehicle. The indicator that, whether, to what extent and/or how far the flow vectors in the background region can be traced back to a movement of the vehicle may be determined depending on the driving speed of the vehicle. By way of example, a weighting between the measure of distance in respect of the mean magnitude and the measure of distance in respect of the measure of homogeneity may be changed in a combined indicator depending on the driving speed of the vehicle. In particular, the weighting may be shifted to the measure of distance in relation to the measure of homogeneity as the driving speed reduces (and vice versa). By taking into account the driving speed during the segmentation, it is possible to further increase the accuracy and robustness of the segmentation.
The separating line may be determined in such a way that the foreground region has a smaller external proportion than the background region. Here, the external proportion may indicate the proportion of a region of the first scene which corresponds to the external portion of the first scene. Alternatively, or in a complementary manner, the external proportion may indicate the proportion of flow vectors which can be traced back to movement of the vehicle.
In accordance with a further aspect, a method for determining a pose of a camera is described. Here, the camera is movably arranged in the interior of a vehicle or at the vehicle (e.g. in smartglasses of the driver of the vehicle). In particular, the camera may be arranged in such a way that a scene captured by the camera comprises parts of the vehicle and parts of the surroundings of the vehicle.
The method comprises capturing a sequence of scenes by use of the camera. Moreover, the method comprises determining a foreground region of a first scene from the sequence of scenes. Here, the foreground region may be determined in accordance with the method described in this document for segmenting a scene. Moreover, the method comprises detecting one or more reference points in the foreground region of the first scene and determining the pose of the camera on the basis of the reference point or reference points detected. By restricting the search of one or more reference points to the foreground region, it is possible to increase the accuracy of the determined pose.
In accordance with one aspect, a control unit configured to carry out one or more of the methods described in this document is described. By way of example, this may be the control unit of an apparatus for augmenting the perception of reality (e.g. smartglasses). Here, the control unit may be configured to determine whether the apparatus for augmenting the perception of reality is situated in a vehicle. If this is the case, it is possible to carry out one of the methods described in this document.
In accordance with a further aspect, an apparatus for augmenting the perception of reality (e.g. smartglasses) comprising the control unit described in this document is described.
In accordance with a further aspect, a vehicle (e.g. an automobile, a truck or a motorbike) comprising the control unit described in this document is described.
In accordance with a further aspect, a software (SW) program is described. The SW program may be configured to be run on a processor (e.g. on a control appliance) in order thereby to carry out the method described in this document.
In accordance with a further aspect, a storage medium is described. The storage medium may comprise a SW program configured to be run on a processor in order thereby to carry out the method described in this document.
It should be noted that the methods, apparatuses and systems described in this document may be used both on their own and in combination with other methods, apparatuses and systems described in this document. Moreover, all aspects of the methods, apparatus and systems described in this document may be combined with one another in multifaceted ways.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.